Who Took Peer Review Seriously: Another Perspective on Student-Generated Quizzes

In educational peer-review activities, one challenge is to tell which peer reviews are credible. Reputation systems are one approach. However, they work best in a scenario where (i) most of the peer reviewers can do a decent job, or (ii) reviewers tend to do peer-reviews in fixed styles─e.g., negative reviewers are negative on all artifacts they review; accurate reviewers grade all work accurately. We argue that, those hypotheses may not hold for many cases for educational peer review. For this reason, we invited student authors in one class to create quiz questions based on their own artifacts. After their reviewers finished peer-review responses, they could take the corresponding quizzes. The mere existence of those quizzes may encourage reviewers to take the peer-review activity more seriously. In addition, their quiz scores can be used as another source of reputation. Since those quizzes are generated by authors to “test” the reviewers, the quiz scores give teaching staff another input on deciding whether a peer review is credible. Our experiments show that a reputation generated from quiz scores provided more accurate estimates of final grades for students’ artifacts than some existing reputation systems.